Skip to content

[perf] Use zmq.asyncio.Context to accelerate notify_data_update process#38

Merged
0oshowero0 merged 5 commits intoAscend:mainfrom
0oshowero0:optimize_manager
Feb 28, 2026
Merged

[perf] Use zmq.asyncio.Context to accelerate notify_data_update process#38
0oshowero0 merged 5 commits intoAscend:mainfrom
0oshowero0:optimize_manager

Conversation

@0oshowero0
Copy link
Collaborator

@0oshowero0 0oshowero0 commented Feb 27, 2026

Description

Problem: The notify_data_update method was declared as async but internally used synchronous zmq.Context and blocking zmq.Poller.poll() / socket.send_multipart() / socket.recv_multipart() calls, which blocked the asyncio event loop and degraded concurrency performance.

Solution:

  • Replace the shared synchronous zmq.Context with zmq.asyncio.Context for the data update notification path.
  • Use await sock.send_multipart() and await asyncio.wait_for(sock.recv_multipart(), ...) for truly non-blocking async I/O.
  • Create per-call dynamic sockets in notify_data_update instead of reusing a long-lived data_status_update_socket, improving isolation and avoiding potential socket state corruption under concurrent calls.
  • Optimize resource lifecycle: handshake socket and its synchronous context are now closed immediately after handshake completes, rather than being held for the entire object lifetime.

Changes

transfer_queue/storage/managers/base.py:

  • Replace long-lived data_status_update_socket with per-call dynamic zmq.asyncio sockets in notify_data_update
  • Replace zmq.Poller-based polling with asyncio.wait_for timeout mechanism

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Copilot AI review requested due to automatic review settings February 27, 2026 02:59
@ascend-robot
Copy link

CLA Signature Pass

0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors the ZMQ socket management in the storage manager to improve performance of the notify_data_update process. It switches from a persistent synchronous socket approach to a dynamic asynchronous socket pattern using zmq.asyncio.Context.

Changes:

  • Replaced persistent data_status_update_socket with dynamic per-request socket creation in notify_data_update
  • Changed from synchronous zmq.Poller to async asyncio.wait_for for receiving ACK responses
  • Separated handshake socket (sync) from data update operations (async) for better resource management

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
@ascend-robot
Copy link

CLA Signature Pass

0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
@ascend-robot
Copy link

CLA Signature Pass

0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
@ascend-robot
Copy link

CLA Signature Pass

0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍

@0oshowero0 0oshowero0 merged commit bd31c02 into Ascend:main Feb 28, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants